Systems and methods for modifying videos based on music

ABSTRACT

Video information defining video content may be accessed. Music information defining a music track providing an accompaniment for video content may be accessed. The music track may have pulses and one or more music events. Individual music events may correspond to different moments within the music track. One or more music events may be individually classified into one or more categories based on intensities of one or pulses occurring within the music event. One or more visual effects may be selected for different moments within the music track based on the categories of the music events. One or more visual effects may be applied to the video content. One or more visual effects may be applied to one or more moments within the video content aligned to one or more different moments within the music track.

FIELD

This disclosure relates to modifying video content based on music thatprovides an accompaniment for the video content.

BACKGROUND

Video editing applications may allow users to manually edit video clipsto introduce visual effects. Music tracks may accompany playback ofvideo clips. Manually editing video clips to introduce visual effectsbased on accompanying music tracks may be time consuming and maydiscourage users from modifying video clips based on music tracks.

SUMMARY

This disclosure relates to modifying videos based on music. Videoinformation defining video content may be accessed. Music informationdefining a music track may be accessed. The music track may provide anaccompaniment for the video content. The music track may have pulses andone or more music events. Individual music events may correspond todifferent moments within the music track. One or more music events maybe individually classified into one or more categories based onintensities of one or pulses occurring within the music event. One ormore visual effects may be selected for different moments within themusic track based on the categories of the music events. One or morevisual effects may be applied to the video content. One or more visualeffects may be applied to one or more moments within the video contentaligned to one or more different moments within the music track.

A system that modifies videos based on music may include one or morephysical processors, and/or other components. The processor(s) may beconfigured by machine-readable instructions. Executing themachine-readable instructions may cause the processor(s) to facilitatemodifying videos based on music. The machine-readable instructions mayinclude one or more computer program components. The computer programcomponents may include one or more of an access component, a visualeffects selection component, a visual effects application component,and/or other computer program components. In some implementations, thecomputer program components may include a music track analysiscomponent.

The access component may be configured to access the video informationdefining one or more video content and/or other information. The accesscomponent may access video information from one or more storagelocations. The access component may be configured to access videoinformation defining one or more video content during acquisition of thevideo information and/or after acquisition of the video information byone or more image sensors.

The access component may be configured to access music informationdefining one or more music tracks and/or other information. The accesscomponent may access music information from one or more storagelocations. The access component may access particular music informationbased on user selection, system information, video content, and/or otherinformation.

A music track may provide an accompaniment for the video content. Themusic track may have pulses. The music track may have one or more musicevents. Individual music events may correspond to different momentswithin the music track. One or more music events may be individuallyclassified into one or more categories based on intensities of one ormore pulses occurring within the music event and/or other information.In some implementations, one or more music events may be classified intoone or more categories based on the intensities of the one or morepulses within a low frequency range, a middle frequency range, and ahigh frequency range. In some implementations, one or more categoriesmay include a weak category, an average category, a strong category, anintense category, and/or other categories.

In some implementations, one or more consecutive pulses within the musictrack may be grouped based on similarity of the intensities within thelow frequency range, the middle frequency range, and the high frequencyrange. In some implementations, one or more consecutive pulses withinthe music track may be grouped based on a hidden Markov model and/orother information.

The music track analysis component may be configured to analyze themusic track to classify the music event(s) within a music track into oneor more categories. The music track analysis component may classify themusic event(s) based on the intensities of one or more pulses occurringwithin the music event and/or other information. The music trackanalysis component may classify the music event(s) based on theintensities of the one or more pulses within a low frequency range, amiddle frequency range, and a high frequency range. In someimplementations, the music track analysis component may classify themusic event(s) into a weak category, an average category, a strongcategory, an intense category, and/or other categories.

The visual effects selection component may be configured to select oneor more visual effects for one or more of the different moments withinthe music track. The visual effects selection component may select oneor more visual effects based on the categories of the one or more musicevents corresponding to the different moments within the music trackand/or other information. In some implementations, the visual effectsselection component may select one or more visual effects based ongrouping(s) of consecutive pulses within the music track. In someimplementations, the visual effects selection component may select oneor more visual effects based on a user selection. In someimplementations, the visual effects selection component may select oneor more visual effects randomly from a list of visual effects.

A visual effect may refer to a change in presentation of the videocontent on a display. A visual effect may change the presentation of thevideo content for a video frame, for multiple frames, for a point intime, and/or for a duration of time. In some implementations, a visualeffect may include one or more changes in perceived speed at which thevideo content is presented during playback. In some implementations, avisual effect may include one or more visual transformation of the videocontent.

The visual effects application component may be configured to apply oneor more visual effects to the video content. The visual effectsapplication component may apply one or more visual effects to one ormore moments within the video content aligned to one or more of thedifferent moments that correspond to the music event(s) within the musictrack. The visual effects application component may apply the visualeffect(s) upon a request for playback of the video content. The visualeffects application component may generate one or more files describingthe visual effects and/or one or more files containing video contentaltered based on the application of the visual effects.

These and other objects, features, and characteristics of the systemand/or method disclosed herein, as well as the methods of operation andfunctions of the related elements of structure and the combination ofparts and economies of manufacture, will become more apparent uponconsideration of the following description and the appended claims withreference to the accompanying drawings, all of which form a part of thisspecification, wherein like reference numerals designate correspondingparts in the various figures. It is to be expressly understood, however,that the drawings are for the purpose of illustration and descriptiononly and are not intended as a definition of the limits of theinvention. As used in the specification and in the claims, the singularform of “a”, “an”, and “the” include plural referents unless the contextclearly dictates otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system that modifies videos based on music.

FIG. 2 illustrates a method for modifying videos based on music.

FIG. 3A illustrates pulses within an example music track.

FIG. 3B illustrates music events within an example music track.

FIGS. 4A-4B illustrate example speed ramps.

FIG. 5A illustrates example speed peaks applied to video content.

FIG. 5B illustrates example speed changes applied to video content.

FIG. 5C illustrates example visual transformations applied to videocontent.

DETAILED DESCRIPTION

FIG. 1 illustrates system 10 for modifying videos based on music. System10 may include one or more of processor 11, electronic storage 12,interface 13 (e.g., bus, wireless interface), and/or other components.Video information 22 defining video content may be accessed by processor11. Music information 24 defining a music track may be accessed. Themusic track may provide an accompaniment for the video content. Themusic track may have pulses and one or more music events. Individualmusic events may correspond to different moments within the music track.One or more music events may be individually classified into one or morecategories based on intensities of one or pulses occurring within themusic event. One or more visual effects may be selected for differentmoments within the music track based on the categories of the musicevents. One or more visual effects may be applied to the video content.One or more visual effects may be applied to one or more moments withinthe video content aligned to one or more different moments within themusic track.

Electronic storage 12 may be configured to include electronic storagemedium that electronically stores information. Electronic storage 12 maystore software algorithms, information determined by processor 11,information received remotely, and/or other information that enablessystem 10 to function properly. For example, electronic storage 12 maystore information relating to video information, video content, musicinformation, music track, pulses, intensities of the pulses, music eventcategories, visual effects, and/or other information.

Electronic storage 12 may store video information 22 defining one ormore video content. Video content may refer to media content that may beconsumed as one or more videos. Video content may include one or morevideos stored in one or more formats/container, and/or other videocontent. A video may include a video clip captured by a video capturedevice, multiple video clips captured by a video capture device, and/ormultiple video clips captured by separate video capture devices. A videomay include multiple video clips captured at the same time and/ormultiple video clips captured at different times. A video may include avideo clip processed by a video application, multiple video clipsprocessed by a video application and/or multiple video clips processedby separate video applications.

Video content may have a progress length. A progress length may bedefined in terms of time durations and/or frame numbers. For example,video content may include a video having a time duration of 60 seconds.Video content may include a video having 1800 video frames. Videocontent having 1800 video frames may have a play time duration of 60seconds when viewed at 30 frames/second. Other time durations and framenumbers are contemplated.

Electronic storage 12 may store music information 24 defining one ormore music tracks. Music tracks may refer to media content that may beconsumed as one or more audios. Music tracks may include recorded music,synthesized music, and/or otherwise produced music. Music track may haveprogress lengths. Progress lengths may be defined in terms of timedurations.

Referring to FIG. 1, processor 11 may be configured to provideinformation processing capabilities in system 10. As such, processor 11may comprise one or more of a digital processor, an analog processor, adigital circuit designed to process information, a central processingunit, a graphics processing unit, a microcontroller, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information. Processor 11 maybe configured to execute one or more machine readable instructions 100to facilitate modifying videos based on music. Machine readableinstructions 100 may include one or more computer program components.Machine readable instructions 100 may include one or more of accesscomponent 102, visual effects selection component 104, visual effectsapplication component 106, and/or other computer program components. Insome implementations, the machine readable instructions 100 may includemusic track analysis component 108.

Access component 102 may be configured to access video informationdefining one or more video content and/or other information. Accesscomponent 102 may access video information from one or more storagelocations. A storage location may include electronic storage 12,electronic storage of one or more image sensors (not shown in FIG. 1),and/or other locations. For example, access component 102 may accessvideo information 22 stored in electronic storage 12. Access component102 may be configured to access video information defining one or morevideo content during acquisition of the video information and/or afteracquisition of the video information by one or more image sensors. Forexample, access component 102 may access video information defining avideo while the video is being captured by one or more image sensors.Access component 102 may access video information defining a video afterthe video has been captured and stored in memory (e.g., electronicstorage 12).

Access component 102 may be configured to access music informationdefining one or more music tracks and/or other information. Accesscomponent 102 may access music information from one or more storagelocations. A storage location may include electronic storage 12,electronic storage of one or more computing devices (not shown in FIG.1), and/or other locations. For example, access component 102 may accessmusic information 24 stored in electronic storage 12.

Access component 102 may access particular music information (e.g.,music information 24) based on user selection, system information, videocontent, and/or other information. For example, access component 102 mayaccess particular music information defining a particular music trackbased on a user's selection of the music track to use as accompanimentfor the video content. Access component 102 may access particular musicinformation defining a particular music track based on a system setting(e.g., last selected music track, next music track, default music track,video summary template specifying a particular music track). Accesscomponent 102 may access particular music information defining aparticular music track based on what is captured (e.g., activity,object, scene, movement, person, emotion) within the video content.Access of music information based on other parameters are contemplated.

A music track may provide an accompaniment for the video content. Amusic track may provide a song, music, and/or other sounds for play withthe playback of one or more portions of or the entire the video content.The music track may be included in a file separate from the filecontaining the video content or may be included in the same file as thevideo content. The music track may be encoded with the video content.

A music track may have pulses. A pulse may refer to the beginning of amusical note or other sounds. In some implementations, pulses may occurat a periodic interval from other pulses. Repetition of pulses atperiodic duration may be perceived as “beats” in a (repeating) series.The pulses that occur at periodic interval may be grouped based onaccents/emphasis of the pulses. For example, pulses consisting of astrong/stressed pulse, a weak/unstressed pulse, and a weak/unstressedpulse may form a pulse group. A reoccurring pulse-group may form a meterof the music track. For example, FIG. 3A illustrates an example musictrack 300. Music track 300 may have pulses 311 that occur at a periodinterval from other pulses. Pulses 311 may include pulses 311A, 311B,311C, 311D, 311E, 311F, 311G, 311H. In some implementations, pulses 311may include strong/stressed pulse of pulse groups. Otherregular/irregular pulses are contemplated.

A music track may have one or more music events. A music event may referto one or more occurrences of particular sound(s) in the music track. Amusic event may refer to occurrences of regular patterns or irregularpatterns of sounds in the music track. A musical event may refer to oneor more changes in the sound(s) in the music track. The particularsound(s)/changes in sound(s) in the music track may be of interest to aviewer/user of the video content with the music track providingaccompaniment. Music events may be determined based on/indicateoccurrence of one of more of a beat, a tempo, a rhythm, an instrument, avolume, a vocal, a chorus, a frequency, a style, a start, an end, and/orother sounds occurring within the music track. For example, referring toFIG. 3B, music event(s) 312 (312A, 312B, 312C, 312D, 312E, 312F, 312G,312H) may refer to occurrences of pulses 311 (311A, 311B, 311C, 311D,311E, 311F, 311G, 311H) within music track 300. Music events may bedetermined based on/indicate occurrence of change in one or more of abeat, a tempo, a rhythm, an instrument, a volume, a vocal, a chorus, afrequency, a style, a start, an end, and/or other sounds occurringwithin the music track. For example, music events may be determinedbased on/indicate occurrence of start/end, parts (e.g., chorus, verse),transitions between parts (e.g., drops/releases/ramps), phrases (e.g.,musical phrases, instruments phrases), hits (e.g., intense percussionsounds related/unrelated to the rhythm grid), bars, beats, strong beats,semi-quaver, quavers, and/or other sounds. Music events may bedetermined based on/indicate occurrence of timing events (e.g., beats,hits) or time-range events (e.g., parts, phrases). In someimplementations, music events may be determined based on user input(e.g., manually specified music event). Other types of music events arecontemplated.

Individual music events may correspond to different moments within themusic track. A moment within the music track may include a point in timewithin the music track or a duration of time within the music track. Amusic event may correspond to a point in time within the music track ora duration of time within the music track. For example, referring toFIG. 3B, one or more of music events 312 may correspond to momentswithin music track 300 corresponding to the occurrences of one or moreof pulses 311. One or more of music events 312 may correspond to momentswithin music track 300 corresponding to occurrences of pulse groups inwhich pulses 311 are strong/stressed pulses.

One or more music events may be individually classified into one or morecategories based on intensities of one or more pulses occurring withinthe music event and/or other information. Intensities of one or morepulses may include one or more of energy and/or amplitude of the pulses.In some implementations, one or more music events may be classified intoone or more categories based on user input (e.g., manual classificationof music events). In some implementations, the pulses may be classifiedinto one or more categories based on their intensities and the musicevents may be classified into one or more categories based on theclassification of the pulses. One or more categories may be associatedwith different values (e.g., ranges) of intensities of the pulses andone or more music events/pulses may be classified based on the values(e.g., individual, summed total) of the intensities of the pulses. Insome implementations, categories of music events/pulses may include aweak category, an average category, a strong category, and an intensecategory. Other categories are contemplated.

In some implementations, one or more music events may be classified intoone or more categories based on the intensities of one or more pulseswithin multiple frequency ranges. In some implementations, multiplefrequency ranges may include a low frequency range, a middle frequencyrange, a high frequency range, and/or other frequency ranges. Forexample, a music track may be converted into a frequency space foranalysis. Intensities of the pulses may be analyzed within multiplefrequency ranges. For example, frequency ranges may include a lowfrequency range between 100-600 Hz; a middle frequency range between1000-5000 Hz; and a high frequency range above 5000 Hz. As anotherexample, frequency ranges may include a sub frequency range between20-40 Hz; a low-end frequency range between 40-160 Hz; a low-midfrequency range between 160-300 Hz; a mid-end frequency range between300-1000 Hz; a mid-high frequency range between 1000-5000 Hz; and ahigh-end frequency range between 5000-20000 Hz. Other frequency rangesare contemplated.

The intensities of the pulses within the individual frequency ranges maybe used to classify the pulses/music events. The intensities of multiplepulses may be combined (e.g., summed, averaged) to determine into whichcategory the pulses/music events will be classified. The categories mayindicate the energy states of the music track/music events based on theanalysis of the pulses.

In some implementations, one or more consecutive pulses within the musictrack may be grouped based on similarity of the intensities withinmultiple frequency ranges (e.g., the low frequency range, the middlefrequency range, and the high frequency range). The similarity of theintensities within the frequency ranges may indicate that theconsecutive pulses correspond to a same part of the music track. In someimplementations, one or more consecutive pulses within the music trackmay be grouped based on a hidden Markov model and/or other information.

In some implementations, one or more music events may be individuallyclassified into one or more categories based on occurrences of othersound(s) within or near the music events. One or more music events maybe classified based on structure of the music track. One or more musicevents may be classified based on whether the music event is within ornear (e.g., at the beginning of/introduces, at the end of/terminates) anintroduction section, a verse section, a pre-chorus section, a chorussection, a refrain section, a bridge section, an instrumental section, asolo section, a conclusion section, and/or other sections. One or moremusic event may be classified based on other characteristics (e.g.,volume, tempo) of sounds that that occur within the music track.

Music track analysis component 108 may be configured to analyze themusic track to classify the music event(s) within a music track into oneor more categories. Music track analysis component 108 may classify themusic event(s) based on the intensities of one or more pulses occurringwithin the music event and/or other information. In someimplementations, the pulses may occur at a periodic interval from otherpulses. Music track analysis component 108 may analyze and quantify theintensities (e.g., energy, amplitude) of the pulses. In someimplementations, music track analysis component 108 may classify themusic event(s) based on the intensities of the one or more pulses withinmultiple frequency ranges (e.g., a low frequency range, a middlefrequency range, and a high frequency range). The intensities ofmultiple pulses may be combined (e.g., summed, averaged) to determineinto which category the pulses/music events may be classified.

In some implementations, music track analysis component 108 may classifythe pulses/music event(s) into a weak category, an average category, astrong category, an intense category, and/or other categories. Othercategories are contemplated. In some implementations, the categorizationof the pulses/music events may be stored in electronic storage (e.g.,electronic storage 12). In some implementations, the categorization ofthe pulses/music events may be stored with the music track or separatelyfrom the music track.

Visual effects selection component 104 may be configured to select oneor more visual effects. A visual effect may refer to a change inpresentation of the video content on a display. A visual effect maychange the presentation of the video content for a video frame, formultiple frames, for a point in time, and/or for a duration of time.

In some implementations, a visual effect may include one or more changesin perceived speed at which the video content is presented duringplayback. For example, the video content may normally played at aperceived speed of 1× (e.g., a video captured at 30 frames/second isdisplayed at 30 frames/second). A visual effect may change the perceivedspeed of the video content playback (e.g., increase perceived speed ofplayback from 1× to 2×; decrease perceived speed of playback from 1× to0.3×) for one or more portions of the video content.

A visual effect that changes the perceived speed of video contentplayback may include one or more speed ramps. A speed ramp may changethe perceived speed of video content playback for a portion of the videocontent and then return the perceived playback speed to the originalperceived playback speed. For example, FIGS. 4A and 4B illustratesexample speed ramps. In FIG. 4A, the speed ramp (speed peak 410) may,for a portion of the video content, increase the perceived playbackspeed above 1× speed and then return the perceived playback speed backto 1× speed. In FIG. 4B, the speed ramp (speed dip 420) may, for aportion of the video content, decrease the perceived playback speedbelow 1× speed and then return the perceived playback speed back to 1×speed. In some implementations, a speed ramp may change the perceivedplayback speed to a value different from the original perceived playbackspeed. Other speed ramps are contemplated.

In some implementations, a visual effect may include one or more visualtransformation of the video content. A visual transformation may includeone or more visual changes in how the video content is presented duringplayback. A visual change may be applied for a moment within theplayback of the video content or for a duration within the playback ofthe video content. In some implementations, a visual transformation mayinclude one or more of a visual zoom, a visual filter, a visualrotation, a visual overlay (e.g., text and/or graphics overlay), and/ora visual vibration (e.g., visual shaking).

Visual effects selection component 104 may select one or more visualeffects for one or more of the different moments within the music track.The different moments within the music track may correspond to differentmusic events. For example, referring to FIG. 3B, the different momentswithin music track 300 for which visual effect(s) are selected maycorrespond to moments corresponding to one or more of music events 312.

Visual effects selection component 104 may select one or more visualeffects based on the categories of the music event(s) corresponding tothe different moment(s) within the music track and/or other information.For example, referring to FIG. 3B, visual effects selection component104 may select one or more visual effects based on categories of musicevents 312. Selecting visual effects based on categories of the musicevents (e.g., 312) may enable visual effects selection component 104 toselect visual effects based on intensities of one or more pulsesoccurring within the music event and/or other information. For example,visual effects selection component 104 may select visual effects basedon intensities of one or more pulses within multiple frequency ranges(e.g., a low frequency range, a middle frequency range, and a highfrequency range).

Music events may form an ordered set—any music event may be comparedwith other music event(s) on an intensity scale. For example, in a giventime range, higher intensity music events may be identified forprovision of visual effects as disclosed herein. In someimplementations, visual effects selection component 104 may distinguishbetween low-level patterns and high-levels patterns of music events andselect different types of visual effects for different levels.

In some implementations, selecting one or more visual effects based oncategories of music events/intensities of pulses may include changingthe amount of impact of the visual effect on the video content. Forexample, one or more visual effects may include changes in perceivedspeed at which the video content is presented during playback, andvisual effects selection component 104 may determine the amount ofchanges in the perceived speed based on the categories of musicevents/intensities of pulses (e.g., larger speed peaks/dips and/or speedchanges based on higher categories/intensities, smaller speed peaks/dipsand/or speed changes based on lower categories/intensities). One or morevisual effects may include visual transformation of the video content,and visual effects selection component 104 may determine the amount ofchanges affected by the visual transformation based on the categories ofmusic events/intensities of pulses (e.g., more dramatic/greater visualchanges based on higher categories/intensities, less dramatic/smallervisual changes based on lower categories/intensities).

In some implementations, visual effects selection component 104 mayselect one or more visual effects based on grouping(s) of consecutivepulses within the music track. For example, referring to FIG. 3B, pulses311 may be grouped into three groups based on similarity of theintensities: pulses 311A, 311B, 311C may be grouped into a high energygroup; pulses 311D, 311E, 311F may be grouped into a low energy group;and pulses 311G, 311H may be grouped into an average energy group.Visual effects selection component 104 may select one or more visualeffects for pulses 311A, 311B, 311C (music events 312A, 312B, 312C)based on its grouping within the high energy group. Visual effectsselection component 104 may select one or more visual effects for pulses311D, 311E, 311F (music events 312D, 312E, 312F) based on its groupingwithin the low energy group. Visual effects selection component 104 mayselect one or more visual effects for pulses 311G, 311H (music events312G, 312H) based on its grouping within the average energy group.

For example, one or more visual effects may include a change in theperceived playback speed of the video content. The visual effectsselected for the high energy group may include increasing the perceivedplayback speed (e.g., above 1× speed). The visual effect selected forthe low energy group may include decreasing the perceived playback speed(e.g., below 1× speed). The visual effect selected for the averageenergy group may include restoring the perceived playback speed (e.g.,to default speed 1×).

As another example, one or more visual effects may include visualtransformation of the video content. The visual effects may be selectedto match the changes in energy state between the groupings of pulses.For example, one or more visual transformation (e.g., mix, dissolve,crossfade, fade, wipe, rotation, shake) may be selected for transitionbetween different energy groups (e.g., between low, average, high,intense). Other selections of visual effects based on groupings arecontemplated.

In some implementations, visual effects selection component 104 mayselect one or more visual effects based on the location of the groupingswithin the music track. For example, visual effects selection component104 may select one or more visual effects based on whether a grouping islocated within/near an introduction section, a verse section, apre-chorus section, a chorus section, a refrain section, a bridgesection, an instrumental section, a solo section, a conclusion section,and/or other sections, and/or whether a group is located within/near thetransition between different sections of the music track. Otherselections of visual effects based on location of groupings arecontemplated.

In some implementations, visual effects selection component 104 mayselect one or more visual effects based on a user selection. A userselection may include selection of visual effect(s) or selection of oneor more criteria for selection of visual effect(s). For example, a usermay select particular visual effects to for particular moments withinthe music track or change the visual effects selected by visual effectsselection component 104 for particular moments within the music track. Auser may select a list of visual effects from which visual effectsselection component 104 may make the selection and/or select when/howvisual effects selection component 104 selects the visual effect(s) orwhen/how the visual effects are triggered.

In some implementations, visual effects selection component 104 mayselect one or more visual effects randomly from a list of visualeffects. For example, visual effects selection component 104 may haveaccess to a list of visual effects and may choose the visual effect(s)at random for different moments within the music track. Visual effectsselection component 104 may remove from the list of visual effectsalready selected visual effects or may reduce the priority ofreselecting already selected visual effects. Such may enable visualeffects selection component 104 to not select the same visual effectrepeatedly for a music track.

In some implementations, visual effects selection component 104 mayselect one or more visual effects based on the duration (e.g., measuredin frames/time) of the video content. For example, visual effectsselection component 104 may select visual transformations rather thanchanges in the perceived playback speed based on the video contenthaving a short duration. Visual effects selection component 104 mayselect changes of perceived playback speeds rather than speed rampsbased on the video content have a long duration. In someimplementations, visual effects selection component 104 may select oneor more visual effects based on the duration of the music track and/orthe duration of a slot within a video summary template into which thevideo content will be placed.

Visual effects application component 106 may be configured to apply oneor more visual effects to the video content. Visual effects applicationcomponent 106 may apply one or more visual effects to one or moremoments within the video content aligned to one or more of the differentmoments that correspond to the music event(s) within the music track.For example, FIG. 5A illustrates example speed peaks 510 applied to thevideo content. Speed peaks 510 may be applied to individual momentswithin the video content aligned to different moments that correspond tomusic events 312A, 312B, 312C, 312D, 312E, 312F, 312G, 312H within musictrack 300.

FIG. 5B illustrates example speed changes 520 applied to the videocontent. Different perceived playback speed within speed changes 520 maybe applied to individual moments within the video content aligned todifferent moments that correspond to music events 312A, 312B, 312C,312D, 312E, 312F, 312G, 312H. For example, based on music events 312A,312B, 312C being categorized in a strong category, perceived playbackspeed greater than 1× speed may be applied to moments within the videocontent aligned to moments corresponding to music events 312A, 312B,312C. Based on music events 312D, 312E, 312F being categorized in a weakcategory, perceived playback speed less than 1× speed may be applied tomoments within the video content aligned to moments corresponding tomusic events 312D, 312E, 312F. Based on music events 312G, 312H beingcategorized in an average category, perceived playback speed of 1× speedmay be applied to moments within the video content aligned to momentscorresponding to music events 312G, 312H.

FIG. 5C illustrates example visual transformations (transition effects530, movement effects 540) applied to the video content. For example,based on changes in energy states of pulses and/or changes in categoriesof music events 312 within music track 300, transition effects 530 maybe applied to individual moments within the video content aligned todifferent moments that correspond to music events 312A, 312D, 312G.Transition effects 530 may include visual effects that emphasizetransitions between energy states of a music track and/or betweendifferent portions (e.g., introduction, verse, chorus) of a music track.For example, transition effects 530 may include one or more of mix,dissolve, crossfade, fade, wipe, and/or other transition effects.

Based on categories of music events 312B, 312C, 312E, 312F, 312H withinmusic track 300, movement effects 540 may be applied to individualmoments within the video content aligned to different moments thatcorrespond to music events 312B, 312C, 312E, 312F, 312H. Movementeffects 540 may include visual effects that emphasize occurrences ofparticular sounds within a music track. For example, movement effects540 may include one or more of rotation, shake, and/or other movementeffects. The magnitude of the movement effects 540 may be determinedbased on the intensities of the pulses/categories of the music events312.

In some implementations, visual effects application component 106 mayapply the visual effect(s) to the video content responsive to a usercommand (e.g., command to create video edits). In some implementations,visual effects application component 106 may apply the visual effect(s)to a preview of the video content (e.g., using lower resolution and/orlower framerate). In some implementations, visual effects applicationcomponent 106 may apply the visual effect(s) to the video contentresponsive to a request for playback of the video content and/or atother times. For example, responsive to the request for playback of thevideo content, visual effects application component 106 may apply thevisual effects during the playback/in real time.

Application of the visual effects to the video content may/may notchange the original file containing the video content. Visual effectsapplication component 106 may generate one or more files describing thevisual effects. For example, visual effects application may generate afile that identifies the visual effects and the portions of the videocontent to which the visual effects are to be applied. Such files may beused at a subsequent time to apply the visual effects to the videocontent. Visual effects application component 106 may generate one ormore files containing video content altered based on the application ofthe visual effects. For example, visual effects component 106 may encodethe video content to include the visual effects (alter the videocontent). Visual effects component 106 may encode the altered videocontent with the music track.

The systems/methods disclosed herein may increase the visual impact ofvideo content and provide for synchronization between audio impact andvideo impact. The types and locations of visual effects may bedetermined based on the content of the music tracks. The systems/methodsdisclosed herein may provide for music driven video time mapping toalter the video content based on what is happening within the musictrack. The visual intensity/look/feel of the video content may bematched to the music track.

In some implementations, video content may include one or more ofspherical video content, virtual reality content, and/or other videocontent. Spherical video content may refer to a video capture ofmultiple views from a single location. Spherical video content mayinclude a full spherical video capture (360 degrees of capture) or apartial spherical video capture (less than 360 degrees of capture).Spherical video content may be captured through the use of one or morecameras/image sensors to capture images/videos from a location. Thecaptured images/videos may be stitched together to form the sphericalvideo content.

Virtual reality content may refer to content that may be consumed viavirtual reality experience. Virtual reality content may associatedifferent directions within the virtual reality content with differentviewing directions, and a user may view a particular directions withinthe virtual reality content by looking in a particular direction. Forexample, a user may use a virtual reality headset to change the user'sdirection of view. The user's direction of view may correspond to aparticular direction of view within the virtual reality content. Forexample, a forward looking direction of view for a user may correspondto a forward direction of view within the virtual reality content.

Spherical video content and/or virtual reality content may have beencaptured at one or more locations. For example, spherical video contentand/or virtual reality content may have been captured from a stationaryposition (e.g., a seat in a stadium). Spherical video content and/orvirtual reality content may have been captured from a moving position(e.g., a moving bike). Spherical video content and/or virtual realitycontent may include video capture from a path taken by the capturingdevice(s) in the moving position. For example, spherical video contentand/or virtual reality content may include video capture from a personwalking around in a music festival.

While the present disclosure may be directed to video content, one ormore other implementations of the system may be configured for othertypes media content. Other types of media content may include one ormore of audio content (e.g., music, podcasts, audio books, and/or otheraudio content), multimedia presentations, images, slideshows, visualcontent (one or more images and/or videos), and/or other media content.

Implementations of the disclosure may be made in hardware, firmware,software, or any suitable combination thereof. Aspects of the disclosuremay be implemented as instructions stored on a machine-readable medium,which may be read and executed by one or more processors. Amachine-readable medium may include any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputing device). For example, a tangible computer readable storagemedium may include read only memory, random access memory, magnetic diskstorage media, optical storage media, flash memory devices, and others,and a machine-readable transmission media may include forms ofpropagated signals, such as carrier waves, infrared signals, digitalsignals, and others. Firmware, software, routines, or instructions maybe described herein in terms of specific exemplary aspects andimplementations of the disclosure, and performing certain actions.

Although processor 11 and electronic storage 12 are shown to beconnected to interface 13 in FIG. 1, any communication medium may beused to facilitate interaction between any components of system 10. Oneor more components of system 10 may communicate with each other throughhard-wired communication, wireless communication, or both. For example,one or more components of system 10 may communicate with each otherthrough a network. For example, processor 11 may wirelessly communicatewith electronic storage 12. By way of non-limiting example, wirelesscommunication may include one or more of radio communication, Bluetoothcommunication, Wi-Fi communication, cellular communication, infraredcommunication, or other wireless communication. Other types ofcommunications are contemplated by the present disclosure.

Although processor 11 is shown in FIG. 1 as a single entity, this is forillustrative purposes only. In some implementations, processor 11 maycomprise a plurality of processing units. These processing units may bephysically located within the same device, or processor 11 may representprocessing functionality of a plurality of devices operating incoordination. Processor 11 may be configured to execute one or morecomponents by software; hardware; firmware; some combination ofsoftware, hardware, and/or firmware; and/or other mechanisms forconfiguring processing capabilities on processor 11.

It should be appreciated that although computer components areillustrated in FIG. 1 as being co-located within a single processingunit, in implementations in which processor 11 comprises multipleprocessing units, one or more of computer program components may belocated remotely from the other computer program components.

The description of the functionality provided by the different computerprogram components described herein is for illustrative purposes, and isnot intended to be limiting, as any of computer program components mayprovide more or less functionality than is described. For example, oneor more of computer program components 102, 104, 106, and/or 108 may beeliminated, and some or all of its functionality may be provided byother computer program components. As another example, processor 11 maybe configured to execute one or more additional computer programcomponents that may perform some or all of the functionality attributedto one or more of computer program components 102, 104, 106, and/or 108described herein.

The electronic storage media of electronic storage 12 may be providedintegrally (i.e., substantially non-removable) with one or morecomponents of system 10 and/or removable storage that is connectable toone or more components of system 10 via, for example, a port (e.g., aUSB port, a Firewire port, etc.) or a drive (e.g., a disk drive, etc.).Electronic storage 12 may include one or more of optically readablestorage media (e.g., optical disks, etc.), magnetically readable storagemedia (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.),electrical charge-based storage media (e.g., EPROM, EEPROM, RAM, etc.),solid-state storage media (e.g., flash drive, etc.), and/or otherelectronically readable storage media. Electronic storage 12 may be aseparate component within system 10, or electronic storage 12 may beprovided integrally with one or more other components of system 10(e.g., processor 11). Although electronic storage 12 is shown in FIG. 1as a single entity, this is for illustrative purposes only. In someimplementations, electronic storage 12 may comprise a plurality ofstorage units. These storage units may be physically located within thesame device, or electronic storage 12 may represent storagefunctionality of a plurality of devices operating in coordination.

FIG. 2 illustrates method 200 for modifying videos based on music. Theoperations of method 200 presented below are intended to beillustrative. In some implementations, method 200 may be accomplishedwith one or more additional operations not described, and/or without oneor more of the operations discussed. In some implementations, two ormore of the operations may occur substantially simultaneously.

In some implementations, method 200 may be implemented in one or moreprocessing devices (e.g., a digital processor, an analog processor, adigital circuit designed to process information, a central processingunit, a graphics processing unit, a microcontroller, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information). The one or moreprocessing devices may include one or more devices executing some or allof the operation of method 200 in response to instructions storedelectronically on one or more electronic storage mediums. The one ormore processing devices may include one or more devices configuredthrough hardware, firmware, and/or software to be specifically designedfor execution of one or more of the operation of method 200.

Referring to FIG. 2 and method 200, at operation 201, video informationdefining video content may be accessed. The video information may bestored in physical storage media. In some implementation, operation 201may be performed by a processor component the same as or similar toaccess component 102 (Shown in FIG. 1 and described herein).

At operation 202, music information defining a music track may beaccessed. The music track may provide an accompaniment for the videocontent. The music track may have pulses and one or more music events.Individual music events may correspond to different moments within themusic track. One or more music events may be individually classifiedinto one or more categories based on intensities of one or more pulsesoccurring within the music event. In some implementations, operation 202may be performed by a processor component the same as or similar toaccess component 102 (Shown in FIG. 1 and described herein).

At operation 203, one or more visual effects may be selected for one ormore of the different moments within the music track. One or more visualeffects may be selected based on the categories of the music event(s).In some implementations, operation 203 may be performed by a processorcomponent the same as or similar to visual effects selection component104 (Shown in FIG. 1 and described herein).

At operation 204, one or more visual effects may be applied to the videocontent. One or more visual effects may be applied to one or moremoments within the video content aligned to one or more of the differentmoments within the music track. In some implementations, operation 204may be performed by a processor component the same as or similar tovisual effects application component 106 (Shown in FIG. 1 and describedherein).

Although the system(s) and/or method(s) of this disclosure have beendescribed in detail for the purpose of illustration based on what iscurrently considered to be the most practical and preferredimplementations, it is to be understood that such detail is solely forthat purpose and that the disclosure is not limited to the disclosedimplementations, but, on the contrary, is intended to covermodifications and equivalent arrangements that are within the spirit andscope of the appended claims. For example, it is to be understood thatthe present disclosure contemplates that, to the extent possible, one ormore features of any implementation can be combined with one or morefeatures of any other implementation.

What is claimed is:
 1. A system for modifying videos based on music, thesystem comprising: one or more physical processors configured bymachine-readable instructions to: access music information defining amusic track, the music track providing an accompaniment for videocontent, the music track having one or more music events, the individualmusic events corresponding to different moments within the music track,wherein the one or more music events are classified based on intensitieswithin multiple frequency ranges of one or more pulses occurring withinthe music event; select one or more visual effects for the one or moremusic events within the music track based on the classification of theone or more music events; and apply the one or more visual effects tothe video content, the one or more visual effects applied to one or moremoments within the video content aligned to the one or more music eventswithin the music track.
 2. The system of claim 1, wherein the one ormore music events are individually classified into one or morecategories, the one or more categories including a weak category and astrong category.
 3. The system of claim 1, wherein one or moreconsecutive pulses are grouped based on similarity of the intensitieswithin the multiple frequency ranges.
 4. The system of claim 3, whereinthe one or more visual effects are selected further based on thegrouping of the consecutive pulses.
 5. The system of claim 1, whereinthe one or more visual effects include one or more changes in aperceived speed at which the video content is presented during playback.6. The system of claim 5, wherein the one or more changes in a perceivedspeed at which the video content is presented during playback includeone or more speed ramps.
 7. The system of claim 1, wherein the one ormore visual effects include one or more visual transformations of thevideo content.
 8. The system of claim 7, wherein the one or more visualtransformations of the video content include one or more of a visualzoom, a visual filter, a visual rotation, a visual overlay, or a visualvibration.
 9. The system of claim 1, wherein the one or more physicalprocessors are further configured by machine-readable instructions toanalyze the music track to classify the one or more music events. 10.The system of claim 1, wherein the multiple frequency ranges include alow frequency range, a middle frequency range, and a high frequencyrange.
 11. A method for modifying videos based on music, the methodcomprising: accessing music information defining a music track, themusic track providing an accompaniment for video content, the musictrack having one or more music events, the individual music eventscorresponding to different moments within the music track, wherein theone or more music events are classified based on intensities withinmultiple frequency ranges of one or more pulses occurring within themusic event; selecting one or more visual effects for the one or moremusic events within the music track based on the classification of theone or more music events; and applying the one or more visual effects tothe video content, the one or more visual effects applied to one or moremoments within the video content aligned to the one or more music eventswithin the music track.
 12. The method of claim 11, wherein the one ormore music events are individually classified into one or morecategories, the one or more categories including a weak category and astrong category.
 13. The method of claim 11, wherein one or moreconsecutive pulses are grouped based on similarity of the intensitieswithin the multiple frequency ranges.
 14. The method of claim 13,wherein the one or more visual effects are selected further based on thegrouping of the consecutive pulses.
 15. The method of claim 11, whereinthe one or more visual effects include one or more changes in aperceived speed at which the video content is presented during playback.16. The method of claim 15, wherein the one or more changes in aperceived speed at which the video content is presented during playbackinclude one or more speed ramps.
 17. The method of claim 11, wherein theone or more visual effects include one or more visual transformations ofthe video content.
 18. The method of claim 17, wherein the one or morevisual transformations of the video content include one or more of avisual zoom, a visual filter, a visual rotation, a visual overlay, or avisual vibration.
 19. The method of claim 11, further comprisinganalyzing the music track to classify the one or more music events. 20.The method of claim 11, wherein the multiple frequency ranges include alow frequency range, a middle frequency range, and a high frequencyrange.